Simulation system, simulation program and simulation method

ABSTRACT

This vehicle synchronization simulation device and means is provided with: a means for calculating positional information of the own vehicle; a means for transmitting the own vehicle positional information to a server means; a means for converting the own vehicle positional information into a specific data format and transmitting the same; a means for transferring the data via a network or a transmission bus inside of a specific device; a means for receiving the data and generating an image; a means for recognizing and detecting a specific object from the generated image; and a means for changing/correcting positional information of the own vehicle using the information resulting from recognition. The server means is provided with a means that synchronously controls three means, a own vehicle position calculating means, a transmitting/receiving means, and an image generating/recognizing means.

CROSS-REFERENCES TO RELATED APPLICATIONS

This Application claims the benefit of priority and is a Continuationapplication of the prior International Patent Application No.PCT/JP2017/033728, with an international filing date of Sep. 19, 2017,which designated the United States, and is related to the JapanesePatent Application No. 2016-197999, filed Oct. 6, 2016 and JapanesePatent Application No. 2017-092949, filed May 9, 2017, the entiredisclosures of all applications are expressly incorporated by referencein their entirety herein.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a simulation system, a simulationprogram and a simulation method of a recognition function module for animage varying with position shifting information of a vehicle.

2. Description of Related Art

At the present time, for the purpose of realizing automatic driving ofvehicles such as ADAS (advanced driver assistance system) or the like todetect and avoid the possibility of an accident in advance, varioustests have actively been conducted by recognizing images of a camerainstalled on a vehicle to detect objects such as other vehicles, walkersand a traffic signal in accordance with an image recognition techniqueto perform control to automatically decrease the speed of the vehicleand avoid the objects and the like. In the case of the above experimentsystem, it is particularly important to synchronously control the entiresystem with a real-time property and a high recognition rate.

An example of an automatic driving support system is a travel controlsystem disclosed, for example, in Patent Document 1. The travel controlsystem disclosed in this Patent Document 1 is aimed at realizing anautomatic driving system with which an vehicle can travel on apredetermined traveling route by detecting road markings such as a lanemarker, a stop position and the like around own vehicle on a road anddetecting solid objects such as a plurality of mobile objects/obstacleslocated around the own vehicle to determine a traveling area on the roadwhile avoiding collision with solid objects such as a traffic signal anda signboard.

Incidentally, for the purpose of performing control by recognizing anoutside peripheral situation with onboard sensors, it is required todetermine vehicles, bicycles and walkers which belong to categories of aplurality of mobile objects and a plurality of obstacles, and detectsinformation about positions and speeds thereof. Furthermore, for drivingown vehicle, it is required to determine the meanings of paints such asa lane marker and a stop sign on a road, and the meanings of trafficsigns. As a vehicle-mounted camera for detecting outside informationaround own vehicle, it is considered effective to use an imagerecognition technique with an image sensor of a camera.

-   [Patent Document] Japanese Unexamined Patent Application Publication    No. 2016-99635

BRIEF SUMMARY OF THE INVENTION

However, while it is practically impossible to collect test data byendlessly driving a vehicle in the actual world, it is an importantissue how to carry out the above test with a sufficient reality of anactually substitutable level.

For example, in the case where an outside environment is recognized byan image recognition technique with camera images, the recognition rateis substantially changed by external factors such as the weather aroundown vehicle (rain, fog or the like) and the time zone (night, twilight,backlight or the like) to influence the detection result. As a result,with respect to mobile objects, obstacles and paints on a load aroundown vehicle, there are increased misdetection and undetection. Suchmisdetection and undetection of an image recognition means can beresolved with a deep leaning (machine learning) technique having ahighest recognition rate by increasing the number of samples forlearning.

However, it has a limit to extract learning samples during actuallydriving on a load, and it is not realistic as a development technique tocarry out a driving test and sample collection after meeting severeweather conditions such as rain, backlight, fog or the like while suchconditions are difficult to reproduce only with a rare opportunity.

On the other hand, for the purpose of realizing fully automatic drivingin future, the above image recognition of camera images would notsuffice. This is because camera images are two-dimensional images sothat, while it is possible to extract objects such as vehicles, walkersand a traffic signal and the like by image recognition,three-dimensional profiles cannot be obtained. Accordingly, a sensorusing a laser beam called LiDAR and a sensor using a millimeter waveband of radio waves are highly anticipated as means for dealing with theabove issues. It is therefore possible to substantially improve thesafety of a vehicle during driving by combining a plurality of differenttypes of sensors.

In order to solve the problem as described above, the present inventionis related to the improvement of the recognition rate of target objectssuch as other vehicles peripheral to own vehicle, obstacles on the road,and walkers, and it is an object of the present invention to improvereality of the driving test of a vehicle and sample collection byartificially generating images which are very similar to actuallyphotographed images taken under conditions, such as severe weatherconditions, which are difficult to reproduce, and provide a simulationsystem, a simulation program and a simulation method which can performsynchronization control with CG images generated by a CG technique bybuilding a plurality of different types of sensors in a virtualenvironment.

In order to accomplish the object as described above, in accordance withthe present invention, a simulation system of a recognition functionmodule for an image varying in correspondence with position shiftinginformation of a vehicle, comprises:

-   -   a positional information acquisition unit which acquires        positional information of the vehicle;    -   an image generation unit which generates a simulation image for        reproducing an area specified by the positional information on        the basis of the positional information acquired by the        positional information acquisition unit;    -   an image recognition unit which recognizes and detects a        particular object by the recognition function module in the        simulation image generated by the image generation unit;    -   a positional information calculation unit which generates a        control signal for controlling behavior of the vehicle by the        use of the recognition result of the image recognition unit, and        changes/modifies the positional information of own vehicle on        the basis of the generated control signal; and    -   a synchronization control unit which controls synchronization        among the positional information acquisition unit, the image        generation unit, the image recognition unit and the positional        information calculation unit.

Alternatively, the present invention is related to a simulator programof a recognition function module for an image varying in correspondencewith position shifting information of a vehicle, causing a computer tofunction as:

-   -   a positional information acquisition unit which acquires        positional information of the vehicle;    -   an image generation unit which generates a simulation image for        reproducing an area specified by the positional information on        the basis of the positional information acquired by the        positional information acquisition unit;    -   an image recognition unit which recognizes and detects a        particular object by the recognition function module in the        simulation image generated by the image generation unit;    -   a positional information calculation unit which generates a        control signal for controlling behavior of the vehicle by the        use of the recognition result of the image recognition unit, and        changes/modifies the positional information of own vehicle on        the basis of the generated control signal; and    -   a synchronization control unit which controls synchronization        among the positional information acquisition unit, the image        generation unit, the image recognition unit and the positional        information calculation unit.

Furthermore, the present invention is related to a simulator method of arecognition function module for an image varying in correspondence withposition shifting information of a vehicle, comprising:

-   -   a positional information acquisition step of acquiring        positional information of the vehicle by a positional        information acquisition unit;    -   an image generation step of generating, by an image generation        unit, a simulation image for reproducing an area specified by        the positional information on the basis of the positional        information acquired in the positional information acquisition        step;    -   an image recognition step of recognizing and detecting a        particular object by the recognition function module in an image        recognition unit in the simulation image generated in the image        generation step;    -   a positional information calculation step of generating a        control signal for controlling behavior of the vehicle by the        use of the recognition result in the image recognition step and        changing/modifying the positional information of own vehicle on        the basis of the generated control signal, by a positional        information calculation unit; and    -   a synchronization control step of controlling synchronization        among the positional information acquisition unit, the image        generation unit, the image recognition unit and the positional        information calculation unit by a synchronization control unit.

In the case of the above invention, it is preferred that thesynchronization control unit comprises:

-   -   a unit of packetizing the positional information in a particular        format and transmitting the packetized positional information;    -   a unit of transmitting the packetized data through a network or        a transmission bus in a particular device;    -   a unit of receiving and depacketizing the packetized data; and    -   a unit of receiving the depacketized data and generating an        image.

In the case of the above invention, the synchronization control unit cantransmit and receive signals among the respective units in accordancewith UDP (User Datagram Protocol).

In the case of the above invention, the positional information of thevehicle includes information about any of XYZ coordinates of roadsurface absolute position coordinates of the vehicle, XYZ coordinates ofroad surface absolute position coordinates of tires, Euler angles of ownvehicle and a wheel rotation angle.

In the case of the above invention, it is preferred that the imagegeneration unit is provided with a unit of synthesizing athree-dimensional profile of the vehicle by computer graphics.

In the case of the above invention, it is preferred that, as the abovevehicle, a plurality of vehicles are set up for each of which therecognition function operates, that

-   -   the positional information calculation unit changes/modifies the        positional information of each of the plurality of vehicles by        the use of information about the recognition result of the        recognition unit, and that    -   the synchronization control unit controls synchronization among        the positional information acquisition unit, the image        generation unit, the image recognition unit and the positional        information calculation unit for each of the plurality of        vehicles.

In the case of the above invention, it is preferred that the imagerecognition unit is a deep learning recognition unit comprising amulti-stage neural network.

In the case of the above invention, it is preferred that the imagerecognition unit further comprises: a segmentation unit which performsarea division of specific objects to be recognized in the simulationimage; and

-   -   a teacher data creation unit which create teacher data for        learning on the basis of the images corresponding to the areas        divided by the segmentation unit.

In the case where the sensors include at least one LiDAR sensor, teacherdata is set up with a plurality of reflectance values as detection itemsfor each object such as a person, a vehicle, a traffic signal or a roadobstacle.

In the case of the above invention, it is preferred that the imagegeneration unit is provided with a unit of generating a different imagefor each sensor unit.

-   -   there is provided, as the sensor unit, with any or all of an        image sensor, a LiDAR sensor, a millimeter wave sensor and an        infrared sensor.

In the case of the above invention, it is preferred that the imagegeneration unit generates 3D graphics images corresponding to aplurality of sensors respectively, that

-   -   the image recognition unit receives the 3D graphics images to        perform deep learning recognition, and outputs a deep learning        recognition result for each sensor, and that    -   the synchronization control unit performs synchronization        control on the basis of a deep learning recognition result of        each sensor.

In the case of the above invention, it is preferred that in the casewhere the sensors include at least one LiDAR sensor, the imagerecognition unit uses, as detection subjects, a plurality of reflectancevalues as detection items for each object such as a person, a vehicle, atraffic signal or a road obstacle.

As has been discussed above, in accordance with the above inventions, itis possible for learning of a recognition function module such as deeplearning (machine learning) to increase the number of samples byartificially generating images such as CG images which are very similarto actually photographed images and improve the recognition rate byincreasing learning efficiency. Specifically, in accordance with thepresent invention, it is possible to artificially and infinitelygenerate images with a light source, an environment and the like whichare do actually not exist by making use of a means for generating and assynthesizing CG images with high reality on the basis of a simulationmodel. Test can be conducted as to whether or not target objects can berecognized and extracted by inputting the generated images to therecognition function module in the same manner as inputting conventionalcamera images, and performing the same process with the generated imagesas with the camera images, and therefore it is possible to performlearning with such types of images as conventionally difficult orimpossible to acquire or take, and furthermore to effectively improvethe recognition rate by increasing learning efficiency.

Furthermore, synergistic effects can be expected by simultaneously usingdifferent types of sensors such as a millimeter wave sensor and a LiDARsensor capable of extracting a three-dimensional profile of an object inaddition to an image sensor capable of acquiring a two-dimensional imageand generating images of these sensors to make it possible to conductextensive tests and perform brush-up of a recognition technique at thesame time.

Incidentally, the application of the present invention covers a widefield, such as, for automatic vehicle driving, experimental apparatuses,simulators, software modules and hardware devices related thereto (forexample, a vehicle-mounted camera, an image sensor, a laser sensor formeasuring a three-dimensional profile of the circumference of avehicle), and machine learning software such as deep learning. Also,since a synchronization control technique is combined with a CGtechnique capable of realistically reproducing actually photographedimage, the present invention can be widely applied to other fields thanthe automatic driving of a vehicle. For example, potential fields ofapplication include a simulator of surgical operation, a militarysimulator and a safety running test system for robot, drone or the like.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic representation showing the overall configurationof the simulator system in accordance with a first embodiment.

FIG. 2 is a schematic representation showing the configuration of adriving simulator device in accordance with the first embodiment.

FIG. 3 is a block diagram for showing the structure of a simulatorserver in accordance with the first embodiment.

FIGS. 4A and 4B are block diagrams for showing the configuration andoperation of image generation and image recognition in accordance withthe first embodiment.

FIG. 5 is a flow chart for showing the procedure of a system inaccordance with the first embodiment.

FIG. 6 is an explanatory view for showing the recognition result ofanother vehicle from a CG image of the system in accordance with thefirst embodiment.

FIG. 7 is an explanatory view for showing the recognition result ofanother vehicle from a CG image of the system in accordance with thefirst embodiment.

FIG. 8 is an explanatory view for showing the recognition result of awalker from a C G image of the system in accordance with the firstembodiment.

FIG. 9 is an explanatory view for showing the general outline of therecognition process of a recognition function module in accordance withthe embodiment.

FIG. 10 is a view for explaining a segmentation map of an image.

FIG. 11 is a view for explaining a segmentation map in which a pluralityof objects on a road are discriminated by colors respectively.

FIG. 12 is a view for explaining a result in which a vehicle and aperson are correctly recognized after deep learning recognition isperformed with a CG image.

FIG. 13 is a view (XML description) for explaining an example ofannotation description in a segmentation map for performing learning ofdeep learning recognition.

FIG. 14 is a view for explaining vehicle synchronization simulationperformed with a plurality of terminals in accordance with a secondembodiment.

FIG. 15 is a view for showing an actual configuration corresponding toFIG. 14.

FIG. 16 is a view for explaining deep learning recognition and behaviorsimulation per formed by generating CG images with a plurality ofsensors in the case of a modification example 1.

FIG. 17 is a view for explaining deep learning recognition and behaviorsimulation per formed by generating CG images with a plurality ofsensors in the case of a modification example 2.

FIG. 18 is a view for explaining the operational mechanism of a LiDAR.

FIGS. 19A to 19C are views for explaining the mechanism of acquiringpoint group data of a LiDAR.

FIG. 20 is a view for explaining beam emission of a LiDAR sensor.

FIG. 21 is a view for explaining the mechanism of acquiringcircumferential point group data generated by a LiDAR.

DETAILED DESCRIPTION OF THE INVENTION First Embodiment

(Overall Configuration of Vehicle Synchronization Simulator System)

In what follows, with reference to the accompanying drawings, anembodiment of a vehicle synchronization simulator system in accordancewith the present invention will be explained in detail. In the case ofthe present embodiment, an example is described in the case where thesimulator system of the present invention is applied to the machinelearning and test of an image recognition function module of a vehicleautomated driving system. In this description, the automated drivingsystem is a system such as ADAS (advanced driver assistance system) orthe like to detect and avoid the possibility of an accident in advance,and performs control to decrease the speed of the vehicle and avoid theobjects and the like by recognizing an image (real image) acquired witha sensor means such as a camera or the like actually mounted on avehicle to detect objects such as other vehicles, walkers and a trafficsignal in accordance with an image recognition technique for the purposeof realizing automatic traveling of vehicles.

FIG. 1 is a schematic representation showing the overall configurationof the simulator system in accordance with the present embodiment. Thesimulator system in accordance with the present embodiment performssimulation programs with respect to a single or a plurality ofsimulation objects, and performs the machine learning and test of thesesimulation programs. As illustrated in FIG. 1, this simulator systemincludes a simulation server 2 located on a communication network 3, andconnected with an information processing terminal 1 a and avehicle-mounted device 1 b for generating or acquiring the position ofown vehicle through the communication network 3.

The communication network 3 is an IP network using the communicationprotocol TCP/IP, and a distributed communication network which isconstructed by connecting a variety of communication lines (a publicnetwork such as a telephone line, an ISDN line, an ADSL line or anoptical line, a dedicated communication line, the third generation (3G)communication system such as WCDMA (registered trademark) and CDMA2000,the fourth generation (4G) communication system such as LTE, the fifthgeneration (5G) or later communication system, and a wirelesscommunication network such as wifi (registered trademark) or Bluetooth(registered trademark)). This IP network includes a LAN such as a homenetwork, an intranet (a network within a company) based on 10BASE-T,100BASE-TX or the like. Alternatively, in many cases, simulator softwareis installed in a PC as the above information processing terminal 1 a.In this case, simulation can be performed by such a PC alone.

The simulator server 2 is implemented with a single server device or agroup of server devices each of which has functions implemented by aserver computer or software capable of performing a variety ofinformation processes. This simulator server 2 includes a servercomputer which executes server application software, or an applicationserver in which is installed middleware for managing and assistingexecution of an application on such a computer.

Furthermore, the simulator server 2 includes a Web server whichprocesses a http response request from a client device. The Web serverperforms data processing and the like, and acts as an intermediary to adatabase core layer in which a relational database management system(RDBMS) is executed as a backend. The relational database server is aserver in which a database management system (DBMS) operates, and hasfunctions to transmit requested data to a client device and anapplication server (AP server) and rewrite or delete data in response toan operation request.

The information processing terminal 1 a and the vehicle-mounted device 1b are client devices connected to the communication network 3, andprovided with arithmetic processing units such as CPUs to provide avariety of functions by running a dedicated client program 5. Thisinformation processing terminal may be implemented with a generalpurpose computer such as a personal computer or a dedicated devicehaving necessary functions, and includes a smartphone, a mobilecomputer, PDA (Personal Digital Assistance), a cellular telephone, awearable terminal device, or the like.

This information processing terminal 1 a or the vehicle-mounted device 1b can access the simulator server 2 through the dedicated client program5 to transmit and receive data. Part or entirety of this client program5 is involved in a driving simulation system and a vehicle-mountedautomated driving system, and executed to recognize real images such asimages captured or detected by a variety of sensors such as avehicle-mounted camera, or captured scenery images (including CG motionpictures in the case of the present embodiment) and the like by the useof an image recognition technique to detect objects such as othervehicles, walkers and a traffic signal in the images, calculate thepositional relationship between own vehicle and the object on the basisof the recognition result, and performs control to decrease the speed ofthe vehicle and avoid the objects and the like in accordance with thecalculation result. Incidentally, the client program 5 of the presentembodiment has the simulator server 2 perform an image recognitionfunction, and calculates or acquires the positional information of ownvehicle by having the own vehicle virtually travel on a map inaccordance with the recognition result of the simulator server 2 orhaving the own vehicle actually travel on the basis of the automaticdriving mechanism of a vehicle positional information calculation unit51 to change the positional information of the own vehicle.

(Configuration of Each Device)

Next, the configuration of each device will specifically be explained.FIG. 2 is a block diagram for showing the internal structure of theclient device in accordance with the present embodiment. FIG. 3 is ablock diagram for showing the internal structure of the simulator serverin accordance with the present embodiment. Meanwhile, in the context ofthis document, the term “module” is intended to encompass any functionunit capable of performing necessary operation, as implemented withhardware such as a device or an apparatus, software capable ofperforming the functionality of the hardware, or any combinationthereof.

(1) Configuration of the Client Device

The information processing terminal 1 a can be implemented with ageneral purpose computer such as a personal computer or a dedicateddevice. On the other hand, the vehicle-mounted device 1 b may be ageneral purpose computer such as a personal computer, or a dedicateddevice (which can be regarded as a car navigation system) such as anautomated driving system. As illustrated in FIG. 2, specifically, theinformation processing terminal 1 a is provided with a CPU 102, a memory103, an input interface 104, a storage device 101, an output interface105 and a communication interface 106. Meanwhile, in the case of thepresent embodiment, these elements are connected to each other through aCPU bus to exchange data thereamong.

The memory 103 and the storage device 101 accumulate data on a recordingmedium, and read out accumulated data from the recording medium inresponse to an request from each device. The memory 103 and the storagedevice 101 may be implemented, for example, by a hard disk drive (HDD),a solid state drive (SSD), a memory card, and the like. The inputinterface 103 is a module for receiving operation signals from anoperation device such as a keyboard, a pointing device, a touch panel orbuttons. The received operation signals are transmitted to the CPU 102so that it is possible to perform operations of an OS or eachapplication. The output interface 105 is a module for transmitting imagesignals and sound signals to output an image and sound from an outputdevice such as a display or a speaker.

Particularly, in the case where the client device is a vehicle-mounteddevice 1 b, this input interface 104 is connected to a system such asthe above ADAS for automatic driving system, and also connected to animage sensor 104 a such as a camera or the like mounted on a vehicle, ora various sensor means such as a LiDAR sensor, a millimeter wave sensor,an infrared sensor or the like which acquires real images, for thepurpose of realizing the automated driving traveling of a vehicle.

The communication interface 106 is a module for transmitting andreceiving data to/from other communication devices on the basis of acommunication system including a public network such as a telephoneline, an ISDN line, an ADSL line or an optical line, a dedicatedcommunication line, the third generation (3G) communication system suchas WCDMA (registered trademark) and CDMA2000, the fourth generation (4G)communication system such as LTE, the fifth (5G) generation or latercommunication system, and a wireless communication network such as wifi(registered trademark) or Bluetooth (registered trademark)).

The CPU 102 is a device which performs a variety of arithmeticoperations required for controlling each element to virtually build avariety of modules on the CPU 102 by running a variety of programs. AnOS (Operating System) is executed and run on the CPU 102 to performmanagement and control of the basic functions of the informationprocessing terminals 1 a to 1 c. Also, while a variety of applicationscan be executed on this OS, the basic functions of the informationprocessing terminal are managed and controlled by running the OS programon the CPU 102, and a variety of function modules are virtually built onthe CPU 102 by running applications on the CPU 102.

In the case of the present embodiment, a client side execution unit 102a is formed by executing the client program 5 on the CPU 102 to generateor acquire the positional information of own vehicle on a virtual map ora real map, and transmit the positional information to the simulatorserver 2. The client side execution unit 102 a receives the recognitionresult of scenery images (including CG motion pictures in the case ofthe present embodiment) obtained by the simulator server 2, calculatethe positional relationship between own vehicle and the object on thebasis of the received recognition result, and performs control todecrease the speed of the vehicle and avoid the objects and the like onthe basis of the calculation result.

(2) Configuration of the Simulator Server

The simulator server 2 in accordance with the present embodiment is agroup of server devices which provide a vehicle synchronizationsimulator service through the communication network 3. The functions ofeach server device can be implemented by a server computer capable ofperforming a variety of information processes or software capable ofperforming the functions. Specifically, as illustrated in FIG. 3, thesimulator server 2 is provided with a communication interface 201, a UDPsynchronization control unit 202, a simulation execution unit 205, a UDPinformation transmitter receiver unit 206, and various databases 210 to213.

The communication interface 201 is a module for transmitting andreceiving data to/from other devices through the communication network 3on the basis of a communication system including a public network suchas a telephone line, an ISDN line, an ADSL line or an optical line, adedicated communication line, the third generation (3G) communicationsystem such as WCDMA (registered trademark) and CDMA2000, the fourthgeneration (4G) communication system such as LTE, the fifth (5G)generation or later communication system, and a wireless communicationnetwork such as wifi (registered trademark) or Bluetooth (registeredtrademark)).

The UDP synchronization control unit 202 is a module for controllingsynchronization between a calculation process to calculate thepositional information of own vehicle by varying the position of the ownvehicle in the client device 1 side, and an image generation process andan image recognition process in the simulator server 2 side. The vehiclepositional information calculation unit 51 of the client device 1acquires the recognition result of an image recognition unit 204 throughthe UDP information transmitter receiver unit 206, generates controlsignals for controlling vehicle behavior by the use of the acquiredrecognition result, changes/modifies the positional information of ownvehicle on the basis of the generated control signals.

The UDP information transmitter receiver unit 206 is a module fortransmitting and receiving data to/from the client side execution unit102 a of the client device 1 in cooperation. In the case of the presentembodiment, the positional information is calculated or acquired in theclient device 1 side, and packetized in a particular format. While thepacketized data is transmitted to the simulator server 2 through anetwork or a transmission bus in a particular device, the packetizeddata is received and depacketized by the simulator server 2, and thedepacketized data is input to an image generation unit 203 to generateimages. Meanwhile, in the case of the present embodiment, the UDPinformation transmitter receiver unit 206 transmits and receives, by theuse of UDP (User Datagram Protocol), signals which are transmitted andreceived among the respective devices with the UDP synchronizationcontrol unit 202.

The above various databases include a map database 210, a vehicledatabase 211 and a drawing database 212. Incidentally, these databasescan be referred to each other by a relational database management system(RDBMS).

The simulation execution unit 205 is a module for generating asimulation image reproducing an area specified on the basis ofpositional information generated or acquired by the positionalinformation acquisition means of the client device 1 and transmitted tothe simulator server 2, and recognizing and detecting particular objectsin the generated simulation image by the use of the recognition functionmodule. Specifically, the simulation execution unit 205 is provided withthe image generation unit 203 and the image recognition unit 204.

The image generation unit 203 is a module for acquiring the positionalinformation acquired or calculated by the positional informationacquisition means of client device 1 and generating a simulation imagefor reproducing, by a computer graphics technique, an area (scenerybased on latitude and longitude coordinates of a map, and direction anda view angle) specified on the basis of the positional information. Thesimulation image generated by this image generation unit 203 istransmitted to the image recognition unit 204.

The image recognition unit 204 is a module for recognizing and detectingparticular objects in the simulation image generated by the imagegeneration unit 203 with the recognition function module 204 a which isunder test or machine learning. The recognition result information D06of this image recognition unit 204 is transmitted to the vehiclepositional information calculation unit 51 of the client device 1. Theimage recognition unit 204 is provided with a learning unit 204 b toperform machine learning of the recognition function module 204 a. Thisrecognition function module 204 a is a module for acquiring a real imageacquired with a sensor means such as a camera device or CG generated bythe image generation unit 203, hierarchically extracting a plurality offeature points in the acquired image, and recognizing objects from thehierarchical combination patterns of the extracted feature points. Thelearning unit 204 b promotes diversification of extracted patterns andimproves learning efficiency by inputting images captured by the abovecamera device or virtual CG images to extract feature points of imageswhich are difficult to image and reproduce in practice.

(Method of the Vehicle Synchronization Simulator System)

The vehicle synchronization simulation method can be implemented byoperating the vehicle synchronization simulator system having thestructure as described above. FIGS. 4A and 4B are block diagrams forshowing the configuration and operation of image generation and imagerecognition in accordance with the present embodiment. FIG. 5 is a flowchart for showing the procedure of a synchronization simulator inaccordance with the present embodiment.

At first, the vehicle positional information calculation unit 51acquires vehicle positional information D02 of own vehicle (S101).Specifically, the client program 5 is executed in the client device 1side to input a various data group D01 such as map information andvehicle initial data to the vehicle positional information calculationunit 51. Next, the positional information of own vehicle on a virtualmap or an actual map is calculated (generated) or acquired by the use ofthe data group D01. The result is transmitted to the simulationexecution unit 205 of the simulator server 2 (S102) as vehiclepositional information D02 through the UDP synchronization control unit202 or the UDP information transmitter receiver unit 206.

Specifically speaking, the vehicle positional information calculationunit 51 transmits the vehicle positional information D02 of own vehicleto the UDP synchronization control unit 202 in accordance with thetiming of a control signal D03 from the UDP synchronization control unit202. Of initial data of the vehicle positional information calculationunit 51, map data, the positional information of own vehicle in the map,the rotation angle and diameter of a wheel of the vehicle body frame andthe like information, can be loaded from the predetermined storagedevice 101. The UDP synchronization control unit 202 and the UDPinformation transmitter receiver unit 206 transmit and receive datafrom/to the client side execution unit 102 a of the client device 1 incooperation. Specifically, the UDP synchronization control unit 202 andthe UDP information transmitter receiver unit 206 transmit the vehiclepositional information D02 calculated or acquired in the client device 1side to the simulator server 2 as packet information D04 packetized in aparticular format with a various data group including vehicleinformation.

While this packetized data is transmitted through a network or atransmission bus in a particular device, the packetized data is receivedand depacketized by the simulator server 2 (S103), and the depacketizeddata D05 is input to the image generation unit 203 of the simulationexecution unit 205 to generate CG images. In this case, the UDPinformation transmitter receiver unit 206 transmits and receives thepacketized packet information D04 of a various data group includingvehicle information among the respective devices by the UDPsynchronization control unit 202 according to UDP (User DatagramProtocol).

Specifically describing, the UDP synchronization control unit 202converts the various data group into the packetized packet informationD04 by UDP packetizing the vehicle positional information D02 of ownvehicle. Thereby, data transmission and reception by the use of the UDPprotocol becomes easy. At this time, UDP (User Datagram Protocol) willbe described to some extent. Generally speaking, while TCP is highreliable and connection oriented and performs windowing control,retransmission control and congestion control, UDP is a connection-lessprotocol which has no mechanism to secure reliability but has asubstantial advantage due to low delay because the process is simple. Inthe case of the present embodiment, since low delay is required duringtransmitting data among the constituent elements, UDP is employedinstead of TCP. Alternatively, RTP (Realtime Transport Protocol) may beused as the most common protocol for voice communication and videocommunication at the present time.

Next, the vehicle positional information D02 of own vehicle specificallycontains, for example, the following information.

-   -   Positional information (three dimensional coordinates (X, Y, Z)        of road surface absolute position coordinates) of own vehicle    -   Euler angles of own vehicle    -   Positional information (three dimensional coordinates (X, Y, Z)        of road surface absolute position coordinates or the like) of        tires    -   Wheel rotation angle    -   Stamping margin of a brake and a steering wheel

Receiving the vehicle positional information D02, the UDP informationtransmitter receiver unit 206 transmits data D05 necessary mainly forgenerating a vehicle CG image, from among information about the vehicle,e.g., XYZ coordinates as the positional information of the vehicle, XYZcoordinates as the positional information of tires, Euler angles andother various information.

Then, the packet information D04 as UDP packets of the various datagroup is divided into a packet header and a payload of a data body by adepacketizing process in the UDP information transmitter receiver unit206. In this case, the UDP packet data can be exchanged by transmissionbetween places remote from each other through a network or transmissioninside a single apparatus such as a simulator through a transmissionbus. The data D05 corresponding to a payload is input to the imagegeneration unit 203 of the simulation execution unit 205 (S104).

In the simulation execution unit 205, the image generation unit 203acquires positional information acquired or calculated by the positionalinformation acquisition means of the client device 1 as the data D05,and generates a simulation image for reproducing, by a computer graphicstechnique, an area (scenery based on latitude and longitude coordinatesof a map, a direction and a view angle) specified on the basis of thepositional information (S105). The 3D graphics composite image generatedby this image generation unit 203 is transmitted to the imagerecognition unit 204.

The image generation unit 203 generates a realistic image by apredetermined image generation method, for example, a CG imagegeneration technique which makes use of the latest physically basedrendering (PBR) technique. The recognition result information D06 isinput to the vehicle positional information calculation unit 51 againand used, e.g., for calculating the positional information of ownvehicle for determining the next behavior of the own vehicle.

The image generation unit 203 generates objects such as a road surface,buildings, a traffic signal, other vehicles and walkers by, for example,a CG technique making use of the PBR technique. This can be understoodas feasible with the latest CG technique from the fact that objects suchas described above are reproduced in a highly realistic manner in atitle of a game machine such as PlayStation. In many cases, objectimages other than own vehicle are stored already as initial data.Particularly, in an automatic driving simulator, a large amount ofsample data such as a number of highways and general roads is stored ina database which can readily be used.

Next, the image recognition unit 204 recognizes and extracts particulartargets, as objects, by the use of the recognition function module 204 awhich is under test or machine learning from simulation images generatedby the image generation unit 203 (S106). In this case, if there is noobject which is recognized (“N” in step S107), the process proceeds tothe next time frame (S109), and the above processes S101 to S107 arerepeated (“Y” in step S109) until all the time frames are processed (“N”in step S109).

On the other hand, if there is an object which is recognized (“Y” instep S107), the recognition result of this image recognition unit 204 istransmitted to the vehicle positional information calculation unit 51 ofthe client device 1 as the recognition result information D06. Thevehicle positional information calculation unit 51 of the client device1 acquires the recognition result information D06 of the imagerecognition unit 204 through the UDP information transmitter receiverunit 206, generates control signals for controlling vehicle behavior bythe use of the acquired recognition result, changes/modifies thepositional information of own vehicle on the basis of the generatedcontrol signals (S108).

Specifically describing, the CG image generated here as the simulationimage D61 is input to the image recognition unit 204 in place of an realimage usually acquired by sensor means and, as already described above,objects are recognized and detected by, for example, a recognitiontechnique such as deep learning. The recognition results as obtained aregiven as area information in a screen (for example, two-dimensional XYcoordinates of an extracted rectangular area) such as other vehicles,walkers, road markings and a traffic signal.

When running a simulator for automatic driving, there are a number ofobjects such as other vehicles, walkers, buildings and a road surface ina screen in which an actual vehicle is moving. Automatic driving isrealized, for example, by automatically turning the steering wheel,stepping on the accelerator, applying the brake and so on whileobtaining realtime information obtained from a camera mounted on thevehicle, a millimeter wave sensor, a radar and other sensors.

Accordingly, in the case of a camera image, a recognition technique suchas deep learning is used to recognize and discriminate objects necessaryfor automatic driving such as other vehicles, walkers, road markings anda traffic signal from among objects displayed on a screen. FIG. 6 andFIG. 7 show an example in which another moving vehicle is recognized andextracted from a CG image G1 of a highway (image G2 of FIG. 7 is ablack-and-white binary image obtained by binarizing the original imageand used for extracting white lines on a road surface. On the otherhand, the upper figure shows a recognition result as a vehicle of animage area similar to the profile of a vehicle). On the other hand, FIG.8 shows an example in which a deep learning technique is used torecognize and detect walkers from a CG image G3. Image areas surroundedby rectangles indicate walkers and can accurately be detected fromplaces near own vehicle and also from places far from the own vehicle.

For example, when another vehicle cuts in front of own vehicle in FIG. 6and FIG. 7, the image recognition unit 204 detects the approach by animage recognition technique, and outputs the recognition resultinformation D06 of the recognition result to the vehicle positionalinformation calculation unit 51. The vehicle positional informationcalculation unit 51 changes the positional information of own vehicle byturning the steering wheel to avoid the another vehicle, applying thebrake to decelerate own vehicle or performing the like operation. In ananother case where a walker of FIG. 8 suddenly runs out in front of ownvehicle, likewise, the vehicle positional information calculation unit51 changes the positional information of own vehicle by turning thesteering wheel to avoid this walker, applying the brake to decelerateown vehicle or performing the like operation.

Meanwhile, in the above described configuration, it is assumed that datais transmitted in a cycle of 25 msec (25 msec is only one example)according to the UDP protocol from the vehicle positional informationcalculation unit 51 to the simulation execution unit 205 through the UDPsynchronization control unit 202 and the UDP information transmitterreceiver unit 206.

Incidentally, the need of “synchronizing model” which is acharacteristic feature of the present invention exists because thevehicle positional information of the next time frame is determined onthe basis of the output result from the simulation execution unit 205 sothat the behavior of a real vehicle cannot be simulated unless theentirety can be synchronously controlled. In the above example,transmission is performed in a cycle of 25 msec. However, ideal delay iszero which is practically impossible, so that UDP is employed to reducethe delay time associated with transmission and reception.

Generally speaking, in the case of an automatic driving simulator, testhas to be conducted with a very large amount of motion image frames. Itis an object of the present embodiment to substitute CG images nearer toactual photographs for an unquestioning amount which cannot be coveredby real driving. Accordingly, it is necessary to guarantee operations inresponse to a long sequence of video sample data.

In the case of the present embodiment, the learning unit 204 bdiversifies extracted pattern to improve learning efficiency byinputting, in addition to images taken by a vehicle mounted cameraduring real driving, virtual CG images generated by the image generationunit 203 to the recognition function module 204 a to extract the featurepoints of images which are difficult to take and reproduce. Therecognition function module 204 a acquires images taken by the cameradevice and CG images, hierarchically extracts a plurality of featurepoints in the acquired images, and recognizes objects on the basis ofcombinational hierarchic patterns of the extracted objects.

FIG. 9 shows the general outline of the recognition process of thisrecognition function module 204 a. As shown in the same figure, therecognition function module 204 a is a multi-class discrimination unitwhich sets up a plurality of objects and detects an object 501 (“person”in this case) including particular feature points from among theplurality of objects. This recognition function module 204 a includesinput units (input layer) 507, first weighting factors 508, hidden units(hidden layer) 509, second weighting factors 510 and output units(output layer) 511.

A plurality of feature vectors 502 is input to the input units 507. Thefirst weighting factors 508 weight the outputs from the input units 507.The hidden unit 509 nonlinearly converts the linear combination of theoutputs from the input units 507 and the first weighting factors 508.The second weighting factors 510 weight the outputs from the hiddenunits 509. The output units 511 calculate the discrimination probabilityof each class (for example, vehicle, walker and motorcycle). In thiscase, while the number of the output units 511 is three, the presentinvention is not limited thereto. The number of the output units 511equals to the number of objects which can be discriminated by thediscrimination unit. By increasing the number of the output units 511,the number of objects which can be discriminated by the discriminationunit can be increased such as two-wheeled vehicle, markings and babybuggy in addition to vehicle, walker and motorcycle.

The recognition function module 204 a in accordance with the presentembodiment is an example of a multi-stage (three-stage in this case)neural network, and the discrimination unit learns the first weightingfactors 508 and the second weighting factors 510 by an error backwardpropagation method. Also, the recognition function module 204 a of thepresent invention is not limited to a neural network, but may be appliedto a multi-layer perceptron and a deep neural network having a pluralityof hidden layers. In this case, the discrimination unit learns the firstweighting factors 508 and the second weighting factors 510 by deeplearning. Also, since the discrimination unit installed in therecognition function module 204 a is a multi-class discrimination unit,for example, it is possible to detect a plurality of objects such asvehicle, walker and motorcycle.

(Outline of Teacher Data Provision Function)

Furthermore, the recognition function module 204 a serves as a deeplearning recognition unit connected to the learning unit 204 b which isprovided with a teacher data creation means for providing teacherlearning data D73 as teacher data for learning as illustrated in FIG.10. Specifically, the teacher data provision function of the learningunit 204 b is implemented with a segmentation unit 71, an annotationgeneration unit 72 and a teacher data creation unit 73.

The segmentation unit 71 is a module for performing area division(segmentation) of specific objects to be recognized in an image toperform deep learning recognition. Specifically, for performing deeplearning recognition, it is necessary in general to perform areadivision of specific objects in an image corresponding to, besides anopposing vehicle, walkers, a traffic signal, guard rail, bicycles,roadside trees and the like which are recognized with a high degree ofaccuracy and at a high speed to realize safe automatic driving.

The segmentation unit 71 performs segmentation of a variety of imagessuch as the simulation image D61 which is a 3D graphics composite imagegenerated by the image generation unit 203 and an actual photographedimage D60 from an existing real image input system. As illustrated inFIG. 11, the segmentation unit 71 generates segmentation images D71 as asegmentation map in which a variety of subject images are discriminatedby colors respectively. The segmentation map is provided with colorinformation for assigning a color to each object (target) as illustratedin the lower part of FIG. 11. For example, grass is green, airplane isred, building is orange, cow is blue, person is ocher, and so on. Also,FIG. 12 shows an example of a segmentation map on a road in which aredisplayed an actual photographed image in the bottom left corner, animage taken by a sensor in the bottom right corner, and segmented areaimages in the intermediate area in which are displayed objects such as aroad in purple, forest in green, obstacles in blue, persons in red andso forth.

The annotation generation unit 72 is a module for performing annotationwhich associates each area image with a particular object. Thisannotation is to furnish relevant information (meta data) as commentarynotes for a particular object associated with each area image, i.e.,describe text by tagging the meta data in a description language such asXML, to separate various information into “meaning of information” and“content of information”. The XML furnished by this annotationgeneration unit 72 is described by associating each object (“content ofinformation” as described above) which is segmented with its information(“meaning of information” as described above, for example, person,vehicle or traffic signal corresponding to each area image).

FIG. 13 shows an image of a certain road reproduced by CG from which areidentified and extracted in square, as a result of a deep learningrecognition technique, vehicle area images (vehicle), person area images(person) which are furnished with annotations. A square can be definedby the XY coordinates of the upper left point and the XY coordinates ofthe lower right point.

The annotation shown in FIG. 13 as an example can be described in anXML, for example, as <all_vehicles>-</all_vehicles> in which isdescribed information about all the vehicles in the image. The firstroad Vehicle-1 is defined as a square area by upper left coordinates of(100, 120) and lower right coordinates of (150, 150). In a like manner,information about all the persons is described in<all_persons>-</all_persons>, and the first road Persons-1 is defined asa square area by upper left coordinates of (200, 150) and lower rightcoordinates of (220, 170).

Accordingly, in the case where there are a plurality of vehicles in theimage, the above description is continued from Vehicle-2 in sequence.Likewise, other objects can be described with tag information, forexample, “bicycle” for bicycle, “signal” for traffic signal and “tree”for tree.

The 3D graphics composite image as the simulation image D61 generated bythe image generation unit 203 is input to the segmentation unit 71 anddivided by the segmentation unit 71 into areas which are distinguishedby color, for example, as illustrated in FIG. 11.

Thereafter, the segmentation images D71 (after distinguished by color)are input to the annotation generation unit 72 which outputs annotationinformation D72 for example described in an XML, description language tothe teacher data creation unit 73. The teacher data creation unit 73attaches tags to the segmentation images D71 and the annotationinformation D72 to create teacher data for deep learning recognition.The tagged teacher learning data D73 becomes the final output result.

Second Embodiment

In what follows, with reference to the accompanying drawings, a secondembodiment of the system in accordance with the present invention willbe explained in detail. FIG. 14 is a schematic representation showing isa view for showing the overall configuration of the system in accordancewith the present embodiment. FIG. 15 is a block diagram for showing theinternal structure of the device in accordance with the presentembodiment. The first embodiment as described above is an embodiment inwhich own vehicle is limited to a single vehicle. The second embodimentis directed to an example in which the positional information of numberof vehicles are simultaneously processed in parallel. Meanwhile, in thedescription of the present embodiment, like reference numbers indicatefunctionally similar elements as the above first embodiment unlessotherwise specified, and therefore no redundant description is repeated.

As shown in FIG. 14, in the case of the present embodiment, a pluralityof client devices 1 c to 1 f are connected to the simulator server 2,and as shown in FIG. 15, while the UDP synchronization control unit 202and the UDP information transmitter receiver unit 206 serve as commonelements in the simulator server 2, in correspondence with the number ofvehicles to be simulated, there are vehicle positional informationcalculation units 51 c to 51 f provided in the client devices 1 c to 1 frespectively and simulation execution units 205 c to 205 f provided inthe simulator server 2.

The vehicle positional information calculation units 51 c to 51 ftransmit vehicle positional information D02 c to D02 f to the UDPsynchronization control unit 202 with the timing of control signals D03c to D03 f. Next, the UDP synchronization control unit 202 converts thevehicle positional information D02 c to D02 f to packet information D04by UDP packetization. Thereby, data transmission and reception by theuse of the UDP protocol becomes easy. The packet information D04 isdivided into a packet header and a payload of a data body by adepacketizing process in the UDP information transmitter receiver unit206. In this case, the UDP packet data can be exchanged by transmissionbetween places remote from each other through a network or transmissioninside a single apparatus such as a simulator through a transmissionbus. The data D05 c to D05 f corresponding to a payload is input to thesimulation execution units 205 c to 205 f.

As has already been discussed above in the first embodiment, thesimulation execution units 205 c to 205 f generates a realistic image bya predetermined image generation method, for example, a CG imagegeneration technique which makes use of the latest physically basedrendering (PBR) technique. The recognition result information D06 c toD06 f is fed back to the vehicle positional information calculationunits 51 c to 51 f to change the position of each vehicle.

Incidentally, while there are four vehicle positional informationcalculation units 51 c to 51 f in the above example, this number is notlimited to four. However, if the number of vehicles to be supportedincreases, synchronization control as a result becomes complicated, andthere is a problem that when there occurs a substantial delay in acertain vehicle, the total delay time increases since the delay times ofthe vehicles are summed up. Accordingly, the configuration can bedesigned in accordance with the hardware scale, processing amount andother conditions of the simulator server.

Incidentally, while PC terminals 1 c to 1 f are remotely connected to avehicle synchronization simulator program 4 through the communicationnetwork 3 in FIG. 14, the PC terminals 1 c to 1 f can be operated in astand-alone manner by installing a program in a local recording mediumsuch as an HDD or an SDD. In this case, there are advantages in thattest can be performed with a low delay and that no influence ofcongestion troubles or the like need not be considered when a shortageof network band is caused.

Furthermore, while 1 c to 1 f are not limited to PC terminals, forexample, when test is conducted with actually moving vehicles, 1 c to 1f can be considered to refer to car navigation systems mounted on thetest vehicles. In this case, rather than recognizing the 3D graphicscomposite image as the simulation image D61 generated by the imagegeneration unit 203 of FIG. 4B, the learning unit 204 receives alive-action video in place of the simulation image D61 so that thesystem can be used for evaluating the performance of the imagerecognition unit 204. This is because, while a human being canimmediately and accurately recognize a walker and a vehicle in alive-action video, it is possible to verify whether or not the imagerecognition unit 204 can output the same result of extraction andrecognition.

MODIFICATION EXAMPLES

Incidentally, the above explanation of the embodiment shows one exampleof the present invention. The present invention is therefore not limitedto the embodiment of the present invention as described above, andvarious modifications and variations are possible in accordance with thedesign and so forth without departing from the spirit of the invention.

Modification Example 1

While the vehicle mounted camera 104 a is a single camera in theexamples of the above embodiments, for example, a plurality of camerasor sensors can be used instead as illustrated in FIG. 16.

A plurality of sensors have to be installed for the purpose of improvingsafety for automatic driving. Accordingly, the recognition rate of anobject in an image can be improved by creating 3D graphics compositeimages from images taken by the use of a plurality of sensors, as inthis modification example, and recognizing the composite images withdeep learning recognition units 61 to 6 n.

In addition, while a plurality of sensors are installed in a singlevehicle in the above second embodiment, it is also possible to recognizecaptured image taken by sensors which are mounted on a plurality ofvehicles moving on a road by a plurality of deep learning recognitionunits in the same manner. Since a plurality of vehicles are often movingat the same time in an actual case, a learning result synchronizationunit 84 synchronizes the recognition results D621 to D62 n of the deeplearning recognition units 61 to 6 n with the same time axis and outputsthe final recognition result as D62.

For example, the 3D graphics composite image as shown in FIG. 13 is animage in which a plurality of vehicles are moving on a road, and thevehicles in the image are generated by a 3D graphics technique. It ispossible to acquire an image from the view point of each individualvehicle by simulatively installing sensors on these vehicles. These 3Dgraphics composite images from the view point of these vehicles areinput to the deep learning recognition units 61 to 6 n to obtainrecognition results.

Modification Example 2

Next, another modification example will be explained which makes use ofa plurality of different types of sensors. While the sensors of theabove modification example 1 are of the same type, for example, the sametype of image sensors, a plurality of different types of sensors areinstalled in this modification example.

As shown in FIG. 17, specifically, a plurality of different types ofvehicle mounted cameras 104 a and 10 b are connected to the inputinterface 104. In this case, the vehicle mounted camera 104 a is a CMOSsensor or a CCD sensor camera which can take an image, in the samemanner as in the above embodiments. On the other hand, the sensor 104 bis a LiDAR (Light Detection and Ranging) which is a device which detectsscattered light of laser radiation emitted in the form of pulses tomeasure the distances of remote objects. The LiDAR has attractedattention as one of indispensable sensors required for increasingprecision of automatic driving.

The sensor 104 b (LiDAR) makes use of near-infrared micropulse light(for example, wavelength of 905 nm) as the laser light, and includes forexample a motor, mirrors and lenses for constructing a scanner and anoptical system. On the other hand, a light receiving unit and a signalprocessing unit of the sensor receive reflected light and calculate adistance by signal processing.

In this case, the LiDAR employs a LiDAR scan device 114 which is calledTOF system (Time of Flight). This LiDAR scan device 114 outputs laserlight as an irradiation pulse Plu1 from a light emitting element 114 bthrough an irradiation lens 114 c on the basis of the control by a laserdriver 114 a as illustrated in FIG. 18. This irradiation pulse Plu1 isreflected by a measurement object Ob1 and enters a light receiving lens114 d as a reflected pulse Plu2, and detected by a light receivingdevice 114 e. The detection result of this light receiving device 114 eis output from the LiDAR scan device 114 as an electrical signal througha signal light receiving circuit 114 f. Such a LiDAR scan device 114emits ultrashort pulses of a rising time of several nano seconds and alight peak power of several tens Watt to an object to be measured, andmeasures the time t required for the ultrashort pulses to reflect fromthe object to be measured and return to the light receiving unit. If thedistance to the object is L and the velocity of light is c, the distanceL is calculated by the following equation.L=(c×t)/2

The basic operation of this LiDAR system is such that, as illustrated inFIGS. 19A to 19C, modulated laser light is emitted from the LiDAR scandevice 114, and reflected by a rotating mirror 114 g, distributed leftand right or rotating by 360° for scanning, and that the laser light asreflected by the object is returned and captured by the LiDAR scandevice 114 again. Finally, the captured reflected light is used toobtain point group data PelY and PelX indicating signal levelscorresponding to rotation angles. Incidentally, for example, the LiDARsystem which is of a rotary type can emit laser light by rotating acenter unit as illustrated in FIG. 20 to performs 360-degree scanning.

Then, in the case of this modification example constructed as describedabove, the simulation image D61 as the 3D graphics synthesized imagebased on the video image taken by the image sensor 104 a such as acamera is a two-dimensional image which is recognized by the deeplearning recognition unit 6.

On the other hand, the point group data acquired by the light emittingelement 114 b is processed by a module additionally provided forprocessing the point group data in the image generation unit 203, whichis provided with a function to generate a 3D point group data graphicimage in the case of the present embodiment.

Then, with respect to the point group data acquired by the sensor 104 b,the image generation unit 203 extracts sensor data acquired by thesensor 104 b, and generates 3D point group data by calculating thedistance to the object by the TOF mechanism with reference to reflectedlight as received on the basis of the extracted sensor data. This 3Dpoint group data corresponds to a so-called distance image based onwhich the 3D point group data is converted to a 3D graphic image.

The 3D point group data graphic image obtained by imaging this 3D pointgroup data may correspond to point group data which is obtained byemitting laser light to all directions of 360 degrees from a LiDARinstalled, for example, on the running center vehicle shown in FIG. 20and FIG. 21 and measuring the reflected light, and the intensity(density) of color indicates the intensity of the reflected light.Incidentally, the area such as a gap in which no substance exists iscolored black because there is no reflected light.

As illustrated in FIG. 21, target objects such as an opposite runningvehicle, a walker and a bicycle can be acquired from actual point groupdata as three-dimensional coordinate data, and therefore it is possibleto easily generate 3D graphic images of these target objects.Specifically, the image generation unit 203 consistently processes pointgroup data to generate a plurality of polygon data items by a 3D pointgroup data graphics image generation function, and 3D graphics can bedrawn by rendering these polygon data items.

Then, the 3D point group data graphic image as generated in this mannerby the image generation unit 203 is input to the deep learningrecognition unit 6 as a simulation image D61, and recognized byrecognition means which has performed learning for 3D point group datain the deep learning recognition unit 6. By this configuration,different means is used than the deep learning recognition means whichhas performed learning with images for image sensors as described in theabove embodiment. As a result, even if an oncoming vehicle is very faraway so that it is likely that the vehicle cannot be acquired by animage sensor, the LiDAR can acquire the size and profile of the oncomingvehicle even at the front of several hundred meters so that therecognition precision can be improved.

As has been discussed above, in accordance with the above modificationexample, there are provided a plurality of sensors having differentcharacteristics or different device properties, and an analysis unit 85can analyze recognition results obtained with the outputs of the sensorsby the deep learning recognition units 61 to 6 n, and output the finalrecognition result D62.

Incidentally, this analysis unit 85 may be arranged outside, forexample, in a network cloud. In this case, even in the case where thenumber of sensors per one vehicle dramatically increases in the futureso that the computational load of the deep learning recognition processincreases, it is possible to improve processing efficiency by performingprocesses, which can be handled outside through a network, by a cloudhaving a large scale computing power and feeding back the results.

Also, in the case of the embodiment of an image sensor, learning isperformed by associating a segmentation map with objects in advance asillustrated in FIG. 11. This corresponds to two-dimensional imagesoutput from the image sensor.

On the other hand, in the case where the sensors include at least oneLiDAR sensor, teacher data is set up with a plurality of reflectancevalues as detection items for each object such as a person, a vehicle, atraffic signal or a road obstacle. Then, for the deep learningrecognition process as described above, in the case where the sensorsinclude at least one LiDAR sensor, the image recognition means detects aplurality of reflectance values for each object such as a person, avehicle, a traffic signal or a road obstacle.

Specifically describing, in the case of a LiDAR sensor,three-dimensional point group data is obtained as illustrated in FIG.21. Accordingly, in the case of a LiDAR sensor, leaning is performed byassociating a three-dimensional point group image with a reflectancevalue of laser light for each object. For example, in the case of theexample shown in FIG. 21, association is set up as a vehicle with areflectance of 1.5 to 2.0, a person on a bicycle with a reflectance of1.1, an adult walker with a reflectance of 0.8, a child walker with areflectance of 0.6, a road with a reflectance of 0.2 to 0.5 and soforth.

Incidentally, while virtual CG images are generated in the case of theexample shown in FIG. 16, as has been discussed in the first embodiment,it is possible to perform deep learning recognition by installing thisapplication system in an actual vehicle (like car navigation system) andinputting information to the system from different types of sensorswhile actually imaging and driving the vehicle. FIG. 17 is a blockdiagram for showing an actual case of such a system.

Also, it is assumed in the case of the present modification example thatthe object imaging devices are a LiDAR sensor and a millimeter wavesensor as described above besides the image sensor of a vehicle mountedcamera. In the case of the image sensor, a high quality CG image isgenerated by a PBR technique as described in the first embodiment withreference to parameters such as light information extracted from aphotographed image as acquired, and the CG image is output from theimage generation unit 203. On the other hand, in the case of the LiDARsensor, a three-dimensional point group data is generated from thereflected light of laser light which is a beam emitted from the LiDARsensor actually mounted on a vehicle. Then, an image as a 3D CGconverted from this three-dimensional point group data is output fromthe above image generation unit 203.

In this way, CG images corresponding to a plurality of types of sensorsare emitted from the image generation unit 203, and the recognitionprocess thereof is performed in each deep learning recognition unit ofFIG. 17 by predetermined means. Also, while the above modificationexample has been explained with a LiDAR sensor as an example, it is alsoeffective to make use of a millimeter wave sensor or an infrared sensorwhich is particularly effective in the night.

EXPLANATION OF SYMBOLS

-   -   B1-B3 . . . building    -   C1, C2 . . . character    -   M1 . . . real map information    -   M11 . . . contour generation image    -   M12 . . . contour line extraction image    -   M2 . . . virtual map information    -   M22, M23 . . . low resolution map    -   M22 a, M23 a . . . boundary area    -   O1-O3 . . . object    -   Ob10 . . . background object    -   W1 . . . real display data    -   W2 . . . virtual display data    -   1 (1 a, 1 b) . . . smartphone    -   2 . . . Internet    -   3 . . . game server    -   10 a, 10 b . . . user    -   11 . . . communication interface    -   12 . . . input interface    -   12 a . . . touch panel    -   13 . . . output interface    -   13 a . . . display    -   14 . . . application running unit    -   15 . . . memory    -   21 . . . satellite    -   22 . . . wireless base station    -   31 . . . communication interface    -   32 . . . positional information management unit    -   33 . . . authentication unit    -   34 . . . game data delivering unit    -   35 a . . . real map database    -   35 b . . . user database    -   35 c . . . game database    -   36 . . . game progress processing unit    -   37 . . . virtual map information management unit    -   141 . . . game progress processing unit    -   142 . . . synchronization processing unit    -   143 . . . event control unit    -   144 . . . positional information acquisition unit    -   145 . . . display control unit    -   146 . . . display data generation unit

What is claimed is:
 1. A simulation system of a plurality of recognitionfunction modules for a real image varying in correspondence withposition shifting information of a plurality of vehicles, comprising: apositional information acquisition unit which acquires, in a pluralityof client devices, a plurality of positional information of theplurality of vehicles; an image generation unit which generates, in asimulator server, a simulation image for reproducing an area specifiedby the plurality of positional information on the basis of the pluralityof positional information acquired by the positional informationacquisition unit; an image recognition unit which recognizes anddetects, in a simulator server, a particular object for each of theplurality of vehicles by the plurality of recognition function modulesin the simulation image generated by the image generation unit; apositional information calculation unit which generates, in theplurality of client devices, a control signal for controlling behaviorof the plurality of vehicles by the use of the recognition result of theimage recognition unit for each of the plurality of vehicles, andchanges/modifies the plurality of positional information of theplurality of vehicles on the basis of the generated control signal; anda synchronization control unit which controls synchronization among thepositional information acquisition unit of the plurality of clientdevices, the image generation unit of the simulator server, the imagerecognition unit of the simulator server and the positional informationcalculation unit of the plurality of client devices.
 2. A simulationsystem of a plurality of recognition function modules for a real imagevarying in correspondence with position shifting information of aplurality of vehicles, comprising: a positional information acquisitionunit which acquires, in a plurality of client devices, a plurality ofpositional information of the plurality of vehicles; an image generationunit which generates, in a simulator server, 3D graphics images forreproducing an area specified by the plurality of positional informationon the basis of the plurality of positional information acquired by thepositional information acquisition unit; an image recognition unit whichrecognizes and detects, in a simulator server, a particular object foreach of the plurality of vehicles by the plurality of recognitionfunction modules in the 3D graphics images generated by the imagegeneration unit to perform deep learning recognition and outputs a deeplearning recognition result for each sensor unit; a positionalinformation calculation unit which generates, in the plurality of clientdevices, a control signal for controlling behavior of the plurality ofvehicles by the use of the recognition result of the image recognitionunit for each of the plurality of vehicles, and changes/modifies theplurality of positional information of the plurality of vehicles on thebasis of the generated control signal; and a synchronization controlunit which controls synchronization among the positional informationacquisition unit of the plurality of client devices, the imagegeneration unit of the simulator server, the image recognition unit ofthe simulator server and the positional information calculation unit ofthe plurality of client devices on the basis of a deep learningrecognition result of each sensor unit, wherein the simulator server isconnected with the plurality of client devices through the communicationnetwork.
 3. The simulation system of claim 1 wherein the synchronizationcontrol unit comprises: a unit of packetizing the plurality ofpositional information in a particular format and transmitting thepacketized plurality of positional information; a unit of transmittingthe packetized data through a network or a transmission bus in aparticular device; a unit of receiving and depacketizing the packetizeddata; and a unit of receiving the depacketized data and generating animage.
 4. The simulation system of claim 1 wherein the synchronizationcontrol unit transmits and receives signals among the respective unitsin accordance with UDP (User Datagram Protocol).
 5. The simulationsystem of claim 1 wherein the plurality of positional information of theplurality of vehicles includes information about any of XYZ coordinatesof road surface absolute position coordinates of the plurality ofvehicles, XYZ coordinates of road surface absolute position coordinatesof tires, Euler angles of the plurality of vehicles and a wheel rotationangle.
 6. The simulation system of claim 1 wherein the image generationunit is provided with a unit of synthesizing a three-dimensional profileof the plurality of vehicles by computer graphics to generate thesimulation image.
 7. The simulation system of claim 1 wherein the imagerecognition unit is a deep learning recognition unit comprising amulti-stage neural network.
 8. The simulation system of claim 1 whereinthe image recognition unit further comprises: a segmentation unit whichperforms area division of specific objects to be recognized in thesimulation image; and a teacher data creation unit which create teacherdata for learning on the basis of the images corresponding to the areasdivided by the segmentation unit.
 9. The simulation system of claim 8wherein in the case where the sensors acquiring the real image includeat least one LiDAR sensor, teacher data is set up with a plurality ofreflectance values as detection items for each object such as a person,a vehicle, a traffic signal or a road obstacle.
 10. The simulationsystem of claim 1 wherein the image generation unit is provided with aunit of generating a different image for each sensor unit acquiring thereal image.
 11. The simulation system of claim 1 wherein there isprovided, as the sensor unit acquiring the real image, with any or allof an image sensor, a LiDAR sensor, a millimeter wave sensor and aninfrared sensor.
 12. The simulation system of claim 2 wherein in thecase where the sensors include at least one LiDAR sensor, the imagerecognition unit uses, as detection subjects, a plurality of reflectancevalues as detection items for each object such as a person, a vehicle, atraffic signal or a road obstacle.
 13. The simulation system of claim 1wherein the simulation image is a virtual CG image generated by a CGtechnique.
 14. The simulation system of claim 1 further comprising: anannotation generation unit which furnishes, in a simulator server, metadata as a commentary note for the particular object detected by theimage recognition unit.